912215 - Server component for dynamic UA override list

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[(inactive) Jim Chen [:jchen] [:darchons]]

Bug 897221 adds a mechanism to dynamically update the UA override list for B2G and Android. Now we need to host the actual list on a server.

Because of the similarity to the addons blocklist, I think we can host the UA override list on addons.mozilla.org alongside the blocklist, using a URL scheme similar to the blocklist. For example,

> https://addons.mozilla.org/ua-override/1/%APP_ID%/%APP_VERSION%/%PRODUCT%/%BUILD_ID%/%CHANNEL%/%OS%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/

The major difference is the UA overrides list returns JSON instead of XML:

> {
>   "yahoo.com":"\\(Mobile#(Android; Mobile",
>   "msn.com":"Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.133 Mobile Safari/535.19"
> }

Either a server script or a file + rewrite rules should work.

Comment 1

[Brad Lassey [:blassey] (use needinfo?)]

Corey, are you the right person to be pinging about this?

Comment 2

[Corey Shields [:cshields]]

(In reply to Brad Lassey [:blassey] (use needinfo?) from comment #1)
> Corey, are you the right person to be pinging about this?

Yes, and we are a bit swamped so I apologize for the delay.  Help me understand some things about this:

* Which company goal/initiative is this supporting?
* Is there a name for this project and URL I can refer to?
* What is the date needed?
* What is the load going to be like?  Are every B2G/Android user going to be hitting this every day?  Is it cachable?

thanks!

Comment 3

[(inactive) Jim Chen [:jchen] [:darchons]]

(In reply to Corey Shields [:cshields] from comment #2)
> (In reply to Brad Lassey [:blassey] (use needinfo?) from comment #1)
> > Corey, are you the right person to be pinging about this?
> 
> Yes, and we are a bit swamped so I apologize for the delay.  Help me
> understand some things about this:
> 
> * Which company goal/initiative is this supporting?
> * Is there a name for this project and URL I can refer to?

It's supporting a new feature in Firefox OS 1.2 and Firefox for Android. I'm not sure if it fits under any goal/initiative. The feature is being called "dynamic UA override", but I don't think it has a wiki/URL yet. I needinfo'd Andreas to see if he has anything to add.

> * What is the date needed?

I think we need this ASAP, because this feature is slated for Firefox OS 1.2 (later this month)

> * What is the load going to be like?  Are every B2G/Android user going to be
> hitting this every day?  Is it cachable?

Most likely each B2G device and Fennec installation will download it once a week. Each download is around 15kB for B2G and <1kB for Fennec. It should be very cacheable due to the static content and infrequent update rate.

Thanks!

Comment 4

[Lawrence Mandel [:lmandel] (use needinfo)]

(In reply to Jim Chen [:jchen :nchen] from comment #3)
> (In reply to Corey Shields [:cshields] from comment #2)
> > (In reply to Brad Lassey [:blassey] (use needinfo?) from comment #1)
> > * Which company goal/initiative is this supporting?

This is in support of Firefox OS. More specifically, this feature supports Web/app content for Firefox OS and partner specific UA/site requirements.

> > * Is there a name for this project and URL I can refer to?
> 
> It's supporting a new feature in Firefox OS 1.2 and Firefox for Android. I'm
> not sure if it fits under any goal/initiative. The feature is being called
> "dynamic UA override", but I don't think it has a wiki/URL yet. I needinfo'd
> Andreas to see if he has anything to add.

This feature supports the mobile Web compatibility effort (https://wiki.mozilla.org/Compatibility/Mobile). It also supports partner specific requirements.

This feature is a replacement for the static UA override mechanism that is currently built into Firefox OS.

Comment 5

[Brandon Burton [:solarce]]

From the original description, it sounds like having this hosted under addons.mozilla.org was a requirement, is that correct?

Comment 6

[(inactive) Jim Chen [:jchen] [:darchons]]

(In reply to Brandon Burton [:solarce] from comment #5)
> From the original description, it sounds like having this hosted under
> addons.mozilla.org was a requirement, is that correct?

Not a hard requirement, but having this alongside the add-ons blocklist seems simpler to maintain.

Comment 7

[Brandon Burton [:solarce]]

(In reply to Jim Chen [:jchen :nchen] from comment #6)
> (In reply to Brandon Burton [:solarce] from comment #5)
> > From the original description, it sounds like having this hosted under
> > addons.mozilla.org was a requirement, is that correct?
> 
> Not a hard requirement, but having this alongside the add-ons blocklist
> seems simpler to maintain.

Adding :oremj / :jason for input on putting it on the AMO infra

Comment 8

[Jake Maul [:jakem]]

A few questions on this... for reference, our team (WebOps) doesn't deal with AMO's blocklist, so I can't use that as a mental reference point... I have no idea how it works. Before we just throw this over the fence to Mark Mayo's team, I'd like to make sure we have a good handle on what's actually going on here, so we can know if that's the right solution. By my read this seems to have very little to do with AMO or Marketplace, it just happens to follow a similar pattern to another app they host (blocklist).


1) Where is the actual code/files/content that need to be hosted?

2) How would this get updated (possibly answered by #1)?

3) How frequent would updates be?

4) Comment 3 says static content, but comment 0 indicates a long URL with lots of "branches". Is this actually a real web app that dynamically generates that content, or a large tree with lots of directories and lots of .json files?

Thanks!

Comment 9

[Wil Clouser [:clouserw]]

5) How is caching/expiration being handled?

Comment 10

[Jeremy Orem [:oremj]]

We should put this on a subdomain, such as, ua-override.addons.mozilla.org, so we have some more flexibility on placement.

It sounds like this is just going to be a static file?

How long can we serve cached data?

Comment 11

[(inactive) Jim Chen [:jchen] [:darchons]]

So the add-ons blocklist is backed by a database and dynamically generated. A copy of the list is periodically checked into m-c and ships with Firefox. The UA override list should behave similarly.

However, in the interest of time (pending B2G release), I think we should use a static file for now, and eventually move to a web app.

(In reply to Jake Maul [:jakem] from comment #8)
> By my read this
> seems to have very little to do with AMO or Marketplace, it just happens to
> follow a similar pattern to another app they host (blocklist).

Yes, having AMO in mind is only because they have experience with a similar app.

> 1) Where is the actual code/files/content that need to be hosted?

For the static file approach, the content will be from a file inside the B2G Gaia repo (I will generate and check in the file ASAP)

> 2) How would this get updated (possibly answered by #1)?

Changes to the list are tracked by individual bugs, so I imagine the list will be updated manually by WebOps for each bug. Or maybe the file can be set up to automatically track the one in the Gaia repo?

> 3) How frequent would updates be?

I'm not sure about the exact frequency. The list is updated on a bug-by-bug basis.

On the user end, each device downloads the list once a week.

> 4) Comment 3 says static content, but comment 0 indicates a long URL with
> lots of "branches". Is this actually a real web app that dynamically
> generates that content, or a large tree with lots of directories and lots of
> .json files?

Since we are using static file for now, it'll be a single file and a rewrite rule to point all those "branches" to the file. When we move to a web app eventually, the URL scheme will stay the same but the script will actually use the parameters.

(In reply to Jeremy Orem [:oremj] from comment #10)
> We should put this on a subdomain, such as, ua-override.addons.mozilla.org,
> so we have some more flexibility on placement.

That sounds good

> It sounds like this is just going to be a static file?

Yes for now, but we will move to a web app eventually.

> How long can we serve cached data?

(In reply to Wil Clouser [:clouserw] from comment #9)
> 5) How is caching/expiration being handled?

Each device downloads the list once a week and saves it locally, so I think caching for a few days will be fine? (but don't take my advice because I don't really know anything about caching :) )

Comment 12

[(inactive) Jim Chen [:jchen] [:darchons]]

The list is now at https://hg.mozilla.org/integration/b2g-inbound/raw-file/tip/b2g/app/ua-update.json.in

and the main copy will be at https://hg.mozilla.org/mozilla-central/raw-file/tip/b2g/app/ua-update.json.in

The server update file will sync to the above one (with comment lines stripped out). Is this simple to do?

Comment 13

[Lawrence Mandel [:lmandel] (use needinfo)]

(In reply to Jim Chen [:jchen :nchen] from comment #12)
> The list is now at
> https://hg.mozilla.org/integration/b2g-inbound/raw-file/tip/b2g/app/ua-
> update.json.in
> 
> and the main copy will be at
> https://hg.mozilla.org/mozilla-central/raw-file/tip/b2g/app/ua-update.json.in

Is there a reason that you selected to host the file in Hg instead of Git, where the rest of Gaia is hosted? Hosting in Git will allow people to continue with their existing tooling.

> The server update file will sync to the above one (with comment lines
> stripped out). Is this simple to do?

Can you elaborate on how the file will be synced? Will this be ongoing? Is there a certain time every week by when changes will need to be in? 

More generally, assuming that we somehow accidentally push a bad update to this file, what is the on device ramification (will invalid prefs in this file or a malformed file break the device in some way) and what is our recovery process?

Comment 14

[Karl Dubost💡 :karlcow]

I basically agree with Lawrence, but go a tad further.

(In reply to Lawrence Mandel [:lmandel] from comment #13)
> (In reply to Jim Chen [:jchen :nchen] from comment #12)
> > The list is now at
> > https://hg.mozilla.org/integration/b2g-inbound/raw-file/tip/b2g/app/ua-
> > update.json.in
> > 
> > and the main copy will be at
> > https://hg.mozilla.org/mozilla-central/raw-file/tip/b2g/app/ua-update.json.in
> 
> Is there a reason that you selected to host the file in Hg instead of Git,
> where the rest of Gaia is hosted? Hosting in Git will allow people to
> continue with their existing tooling.

Be Gaia or Mozilla-Central it makes it a bit harder to participate. There's a friction effect with regards to this. For anyone who would have to manage this file (currently Lawrence or me), it requires to download the full project it belongs too and keep in sync with the updates of the full project. That means that you need a preinitialization of a few 100s Mo and then to update a few Mo every week. 

All of that for updating a file which is just a few Ko and where the modifications are on the basis of a few octets. It doesn't seem to be effective. Should the full process of updating in a separate repo and merge automatically in whatever repo it is required. 

On Opera browsers (for browserJS) there are three options:

1. On download automagically once a week. If the browser has not been online for a long time, it will check if there's a new version. So it requires to check the internal date of the file, to keep memory of the last time it has been downloaded and to update if more than x days.
2. Deactivate the UA override. Which answers for example the need of Web developers when they test their own Web site and QA department of Opera.
3. Force reload. Which is basically even if you have an updated list, the browser will force the download of the list. This option is a one time switch, it goes back to 1. automatically so not to stay in a permanent state.

Comment 15

[(inactive) Jim Chen [:jchen] [:darchons]]

(In reply to Lawrence Mandel [:lmandel] from comment #13)
> 
> Is there a reason that you selected to host the file in Hg instead of Git,
> where the rest of Gaia is hosted? Hosting in Git will allow people to
> continue with their existing tooling.

The user of this file is a JS component in Gecko, and it makes things more streamlined to put the file in Gecko. It lets us package and ship the list inside Gecko.

Here's my post to the compat and dev-gaia lists, https://groups.google.com/forum/#!topic/mozilla.compatibility/5L5mYtiemM8

> > The server update file will sync to the above one (with comment lines
> > stripped out). Is this simple to do?
> 
> Can you elaborate on how the file will be synced? Will this be ongoing? Is
> there a certain time every week by when changes will need to be in? 

We will have to figure it out with WebOps, but it could be every day, every week, after every change, etc.

> More generally, assuming that we somehow accidentally push a bad update to
> this file, what is the on device ramification (will invalid prefs in this
> file or a malformed file break the device in some way) and what is our
> recovery process?

Right now there's a bug that will make invalid updates disable overrides entirely; I will fix that bug. The intended behavior is to keep using the previously saved copy.

(In reply to Karl Dubost :karlcow from comment #14)
> All of that for updating a file which is just a few Ko and where the
> modifications are on the basis of a few octets. It doesn't seem to be
> effective. Should the full process of updating in a separate repo and merge
> automatically in whatever repo it is required. 

This is a short-term solution for the B2G 1.2 release, but I think next we want to move to an auto-checkin system where the repo's copy is synced to an external source (similar to how the addons blocklist works right now).

> On Opera browsers (for browserJS) there are three options:
> 
> 1. On download automagically once a week. If the browser has not been online
> for a long time, it will check if there's a new version. So it requires to
> check the internal date of the file, to keep memory of the last time it has
> been downloaded and to update if more than x days.

The update system already does this.

> 2. Deactivate the UA override. Which answers for example the need of Web
> developers when they test their own Web site and QA department of Opera.

This can already be done through pref.

> 3. Force reload. Which is basically even if you have an updated list, the
> browser will force the download of the list. This option is a one time
> switch, it goes back to 1. automatically so not to stay in a permanent state.

Can already be done through pref too.

Comment 16

[Lawrence Mandel [:lmandel] (use needinfo)]

(In reply to Jim Chen [:jchen :nchen] from comment #15)
> The user of this file is a JS component in Gecko, and it makes things more
> streamlined to put the file in Gecko. It lets us package and ship the list
> inside Gecko.
> 
> Here's my post to the compat and dev-gaia lists,
> https://groups.google.com/forum/#!topic/mozilla.compatibility/5L5mYtiemM8

Thanks for the pointer to your mailing list post. Sorry that I missed that earlier. I'm ok with keeping the file in Hg.

> We will have to figure it out with WebOps, but it could be every day, every
> week, after every change, etc.

Are one of these options in place now? If so, let's work out the other details. We can revisit this later.
 
> Right now there's a bug that will make invalid updates disable overrides
> entirely; I will fix that bug. The intended behavior is to keep using the
> previously saved copy.

Great. Bug #?

> > 2. Deactivate the UA override. Which answers for example the need of Web
> > developers when they test their own Web site and QA department of Opera.
> 
> This can already be done through pref.

What's the pref? Is this documented somewhere?
 
> > 3. Force reload. Which is basically even if you have an updated list, the
> > browser will force the download of the list. This option is a one time
> > switch, it goes back to 1. automatically so not to stay in a permanent state.
> 
> Can already be done through pref too.

Same questions as above.

Comment 17

[Karl Dubost💡 :karlcow]

So I guess I need to adjust and revise the process, which hopefully will make it easier.
https://wiki.mozilla.org/Compatibility/Mobile/WipeOutUAOverides#Updating_the_UA_override_list

Could someone explain a bit what is the process for doing pull request on hg.mozilla.org :)

Comment 18

[Lawrence Mandel [:lmandel] (use needinfo)]

See https://developer.mozilla.org/en-US/docs/Developer_Guide/Source_Code

Comment 19

[(inactive) Jim Chen [:jchen] [:darchons]]

(In reply to Lawrence Mandel [:lmandel] from comment #16)
> 
> > We will have to figure it out with WebOps, but it could be every day, every
> > week, after every change, etc.
> 
> Are one of these options in place now? If so, let's work out the other
> details. We can revisit this later.

Not yet. WebOps still has to set up the server.

> > Right now there's a bug that will make invalid updates disable overrides
> > entirely; I will fix that bug. The intended behavior is to keep using the
> > previously saved copy.
> 
> Great. Bug #?

Bug 917965; under review and will be uplifted to 1.2.

> > > 2. Deactivate the UA override. Which answers for example the need of Web
> > > developers when they test their own Web site and QA department of Opera.
> > 
> > This can already be done through pref.
> 
> What's the pref? Is this documented somewhere?

Where should the docs go?

Setting "general.useragent.updates.enabled" to false will disable updating overrides and also disable applying updated overrides (overrides through prefs are independent and are not affected)

> > > 3. Force reload. Which is basically even if you have an updated list, the
> > > browser will force the download of the list. This option is a one time
> > > switch, it goes back to 1. automatically so not to stay in a permanent state.
> > 
> > Can already be done through pref too.
> 
> Same questions as above.

Forcing an update can be done by setting "general.useragent.updates.interval" to a small value like 30 (seconds), and then restoring the setting later.

Comment 20

[Karl Dubost💡 :karlcow]

(In reply to Jim Chen [:jchen :nchen] from comment #19)
> Forcing an update can be done by setting
> "general.useragent.updates.interval" to a small value like 30 (seconds), and
> then restoring the setting later.

Automatically or manually. I would encourage to do it automatically. Because it is very likely people will forget to restore the setting and then the server will be slowly but steadily hammered by devices which have changed the setting.

Comment 21

[Lawrence Mandel [:lmandel] (use needinfo)]

(In reply to Jim Chen [:jchen :nchen] from comment #19)
> (In reply to Lawrence Mandel [:lmandel] from comment #16)
> > > > 2. Deactivate the UA override. Which answers for example the need of Web
> > > > developers when they test their own Web site and QA department of Opera.
> > > 
> > > This can already be done through pref.
> > 
> > What's the pref? Is this documented somewhere?
> 
> Where should the docs go?
> 
> Setting "general.useragent.updates.enabled" to false will disable updating
> overrides and also disable applying updated overrides (overrides through
> prefs are independent and are not affected)

Karl - Can you please include these prefs somewhere in the compat docs.

Sheppy - Do we have the Firefox OS specific preferences documented somewhere? (Note that this is what I would call a developer pref as opposed to something that we should expose to a typical end user.)

> > > > 3. Force reload. Which is basically even if you have an updated list, the
> > > > browser will force the download of the list. This option is a one time
> > > > switch, it goes back to 1. automatically so not to stay in a permanent state.
> > > 
> > > Can already be done through pref too.
> > 
> > Same questions as above.
> 
> Forcing an update can be done by setting
> "general.useragent.updates.interval" to a small value like 30 (seconds), and
> then restoring the setting later.

Ditto for documenting this pref.

Comment 22

[Eric Shepherd [:sheppy]]

We have this page documenting Firefox OS settings:

https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Platform/Settings_list

Comment 23

[Jake Maul [:jakem]]

Some thoughts on this...

Given that the primary downloaders will be FirefoxOS phones, we can probably safely assume that *initial* usage will be small (compared to something like Snippets, which desktop Firefox relies on), but growing over time as FirefoxOS usage grows.

The relatively static nature of this content, coupled with the long turnaround on client-side fetching (once per week) makes it a good candidate for a CDN. We can set a 24-hour TTL and (more or less) forget about the end-user-delivery side of the app.

For updating, as long as it's just one file, we should be able to sync with the current copy in Hg relatively easily... we can't really pull down all of mozilla-central to get it or anything, but we can probably do something really simple like "curl https://hg.mozilla.org/mozilla-central/raw-file/tip/b2g/app/ua-update.json.in" in a cron job. my concern with that would be, how do we know this file is good to pull at any given time? It's mozilla-central after all, and we're pushing it out to production FirefoxOS devices. That makes me wonder if it needs to be manually triggered instead of automatic via cron.


My next question is, what happens if something goes wrong? Specifically on 2 fronts:

1) How does the browser validate that the contents received are complete/correct/usable? For instance, let's say there's a bad checkin and the web nodes get bad data. Or maybe there's some sort of corruption between Hg and the web nodes, or the web nodes and the CDN, or the CDN and the end user. What happens?

2) What happens if the service is offline / unreachable? For example, in the case of the Snippets service, Firefox has a set of built-in snippets that it will display instead. AUS will try again in 12 hours. For Telemetry, FHR, and Socorro, I believe that upload packet is simply discarded (though I could be wrong). How will FirefoxOS react to this service being unavailable?


The answers to these will dictate how extreme we have to be in supporting this (one VM in one datacenter, vs massive redundancy and availability guarantees). If FirefoxOS is highly tolerant of these sorts of issues, we can have something up and running pretty easily.

Thanks!

Comment 24

[(inactive) Jim Chen [:jchen] [:darchons]]

(In reply to Jake Maul [:jakem] from comment #23)
> Some thoughts on this...
> 
> Given that the primary downloaders will be FirefoxOS phones, we can probably
> safely assume that *initial* usage will be small (compared to something like
> Snippets, which desktop Firefox relies on), but growing over time as
> FirefoxOS usage grows.

Right. I think it'd be good to keep a tab on the number of downloads over time.

> My next question is, what happens if something goes wrong? Specifically on 2
> fronts:
> 
> 1) How does the browser validate that the contents received are
> complete/correct/usable? For instance, let's say there's a bad checkin and
> the web nodes get bad data. Or maybe there's some sort of corruption between
> Hg and the web nodes, or the web nodes and the CDN, or the CDN and the end
> user. What happens?

The JSON syntax is checked as part of requesting the file. Invalid files are rejected and the last-known-good version will be used. Beyond syntax, we don't specifically verify the data. I think for UA overrides, the worst that corrupt data will do is to make a website return a desktop site instead of a mobile site; although it's undesirable, I don't think it's disruptive, and a week turnaround for a new update/fix seems reasonable. Therefore, I think we can tolerate some bad data. Changes to the file are also reviewed like other patches, so hopefully there won't be many bad checkins.

> 2) What happens if the service is offline / unreachable? For example, in the
> case of the Snippets service, Firefox has a set of built-in snippets that it
> will display instead. AUS will try again in 12 hours. For Telemetry, FHR,
> and Socorro, I believe that upload packet is simply discarded (though I
> could be wrong). How will FirefoxOS react to this service being unavailable?

On the device, we keep both a shipped version of the file and a last-known-good downloaded version. If the update is unreachable, instead of a week, we will attempt another update the next day. In the meantime, we would keep using the saved version. So I think the system is fairly tolerant in terms of redundancy and availability.

Thanks!

Comment 25

[Jake Maul [:jakem]]

Is this what you're looking for? (Just a proof-of-concept... don't embed this URL anywhere)

http://dynamic-ua-override.paas.allizom.org/ua-update.json-in

We can have that done this week... should be extremely simple. Maybe today, though I wouldn't bank on it.

Comment 26

[(inactive) Jim Chen [:jchen] [:darchons]]

(In reply to Jake Maul [:jakem] from comment #25)
> Is this what you're looking for? (Just a proof-of-concept... don't embed
> this URL anywhere)
> 
> http://dynamic-ua-override.paas.allizom.org/ua-update.json-in
> 
> We can have that done this week... should be extremely simple. Maybe today,
> though I wouldn't bank on it.

That looks great! Only thing is we want to strip out the comment lines (//) in the file because they are not valid JSON. We do this when building B2G [1] and we should do it here, too. And the result should be "ua-update.json" without the "-in" bit. Thank you!

[1] http://mxr.mozilla.org/mozilla-central/source/b2g/app/Makefile.in#71

Comment 27

[Brandon Burton [:solarce]]

(In reply to Jim Chen [:jchen :nchen] from comment #26)
> (In reply to Jake Maul [:jakem] from comment #25)
> > Is this what you're looking for? (Just a proof-of-concept... don't embed
> > this URL anywhere)
> > 
> > http://dynamic-ua-override.paas.allizom.org/ua-update.json-in
> > 
> > We can have that done this week... should be extremely simple. Maybe today,
> > though I wouldn't bank on it.
> 
> That looks great! Only thing is we want to strip out the comment lines (//)
> in the file because they are not valid JSON. We do this when building B2G
> [1] and we should do it here, too. And the result should be "ua-update.json"
> without the "-in" bit. Thank you!
> 

Is this something that the team committing the file to VCS should be doing or something you're asking be done as part of putting a copy of the file on the server(s)? 

> [1] http://mxr.mozilla.org/mozilla-central/source/b2g/app/Makefile.in#71

Comment 28

[(inactive) Jim Chen [:jchen] [:darchons]]

(In reply to Brandon Burton [:solarce] from comment #27)
> (In reply to Jim Chen [:jchen :nchen] from comment #26)
> > 
> > That looks great! Only thing is we want to strip out the comment lines (//)
> > in the file because they are not valid JSON. We do this when building B2G
> > [1] and we should do it here, too. And the result should be "ua-update.json"
> > without the "-in" bit. Thank you!
> > 
> 
> Is this something that the team committing the file to VCS should be doing
> or something you're asking be done as part of putting a copy of the file on
> the server(s)? 
> 
> > [1] http://mxr.mozilla.org/mozilla-central/source/b2g/app/Makefile.in#71

The Hg version will always have the comment lines, but the comment lines should be stripped out when copying from Hg to the file server (or when serving the file).

Comment 29

[Brandon Burton [:solarce]]

I've deployed the beginning of the production version of this to dynamicua.cdn.mozilla.net, once we're happy with the on-server setup, we'll move the live DNS to go through our CDN for performance and resilience, but the backend is a clustered web server setup

I've written a small shell script which downloads the file from Mercurial and strips out the comments using sed and then does verification of the .json, you can see a copy at https://gist.github.com/solarce/43d3d63111f4713b413b

I'll document the current setup at https://mana.mozilla.org/wiki/display/websites/dynamicua.cdn.mozilla.net in the morning

Let me know how things look and if there are any questions

Comment 30

[Brandon Burton [:solarce]]

(In reply to Brandon Burton [:solarce] from comment #29)
> I've deployed the beginning of the production version of this to
> dynamicua.cdn.mozilla.net, once we're happy with the on-server setup, we'll
> move the live DNS to go through our CDN for performance and resilience, but
> the backend is a clustered web server setup
> 

Once it's pointed to the CDN we'll have SSL support

I made dynamicua.cdn.mozilla.net return ua-update.json as the "index" file but that behavior can be changed if needed

Comment 31

[(inactive) Jim Chen [:jchen] [:darchons]]

Awesome! It just occurred to me, but can we have a URL that uses the following format?

> dynamicua.cdn.mozilla.net/0/{3c2e2abc-06d4-11e1-ac3b-374f68613e61}

0 is a version number, for when we may support additional query parameters in the future

{3c2e2abc-06d4-11e1-ac3b-374f68613e61} is the B2G app ID, so that we can support other products such as Fx for Android in the future.

Thanks!

Comment 32

[Brandon Burton [:solarce]]

(In reply to Jim Chen [:jchen :nchen] from comment #31)
> Awesome! It just occurred to me, but can we have a URL that uses the
> following format?
> 
> > dynamicua.cdn.mozilla.net/0/{3c2e2abc-06d4-11e1-ac3b-374f68613e61}
> 
> 0 is a version number, for when we may support additional query parameters
> in the future
> 

Well, we could do something hacky with /VERSION/ being an actual folder, if you want it to be something dynamic we'd need to get someone who can write a small web application to help with that

> {3c2e2abc-06d4-11e1-ac3b-374f68613e61} is the B2G app ID, so that we can
> support other products such as Fx for Android in the future.

Potentially, but how would we obtain the app ID for use in the URL/filename?

Comment 33

[(inactive) Jim Chen [:jchen] [:darchons]]

(In reply to Brandon Burton [:solarce] from comment #32)
> (In reply to Jim Chen [:jchen :nchen] from comment #31)
> > Awesome! It just occurred to me, but can we have a URL that uses the
> > following format?
> > 
> > > dynamicua.cdn.mozilla.net/0/{3c2e2abc-06d4-11e1-ac3b-374f68613e61}
> > 
> > 0 is a version number, for when we may support additional query parameters
> > in the future
> > 
> 
> Well, we could do something hacky with /VERSION/ being an actual folder, if
> you want it to be something dynamic we'd need to get someone who can write a
> small web application to help with that

I think a directory is fine for now, and can be replaced later if needed.

> > {3c2e2abc-06d4-11e1-ac3b-374f68613e61} is the B2G app ID, so that we can
> > support other products such as Fx for Android in the future.
> 
> Potentially, but how would we obtain the app ID for use in the URL/filename?

The B2G app ID is always {3c2e2abc-06d4-11e1-ac3b-374f68613e61}. On the client side it's part of the request URL, and on the server side I think can just name the file "{3c2e2abc-06d4-11e1-ac3b-374f68613e61}".

If we do support Fx for Android later, it'd be another cron job that saves to another file with the Fx for Android app ID as its name.

Comment 34

[(inactive) Jim Chen [:jchen] [:darchons]]

Any update, Brandon?

Comment 35

[Brandon Burton [:solarce]]

(In reply to Jim Chen [:jchen :nchen] from comment #34)
> Any update, Brandon?

I'm working on implementing the changes discussed above and should have them ready to test in a couple hours

Comment 36

[Brandon Burton [:solarce]]

http://dynamicua.cdn.mozilla.net/0/%7B3c2e2abc-06d4-11e1-ac3b-374f68613e61%7D is now working as expected and https://mana.mozilla.org/wiki/display/websites/dynamicua.cdn.mozilla.net is current with documentation

I am working on configuring the CDN pieces now, as we want that before we go into production

:nchen, What do you want the workflow for the .json being updated to be? Should it run from a cron job?

Comment 37

[(inactive) Jim Chen [:jchen] [:darchons]]

(In reply to Brandon Burton [:solarce] from comment #36)
> http://dynamicua.cdn.mozilla.net/0/%7B3c2e2abc-06d4-11e1-ac3b-
> 374f68613e61%7D is now working as expected and
> https://mana.mozilla.org/wiki/display/websites/dynamicua.cdn.mozilla.net is
> current with documentation
> 
> I am working on configuring the CDN pieces now, as we want that before we go
> into production
> 
> :nchen, What do you want the workflow for the .json being updated to be?
> Should it run from a cron job?

Awesome! Thank you very much! A cron job is fine, maybe once a week on Sunday? We can revisit this later if needed.

Comment 38

[Brandon Burton [:solarce]]

(In reply to Jim Chen [:jchen :nchen] from comment #37)

> Awesome! Thank you very much! A cron job is fine, maybe once a week on
> Sunday? We can revisit this later if needed.

That works fine, to be clear, the "version" in the URL, currently "0" should only be changed by hand though, right? Or should it increment each time there is a new revision of the file?

Comment 39

[(inactive) Jim Chen [:jchen] [:darchons]]

(In reply to Brandon Burton [:solarce] from comment #38)
> (In reply to Jim Chen [:jchen :nchen] from comment #37)
> 
> > Awesome! Thank you very much! A cron job is fine, maybe once a week on
> > Sunday? We can revisit this later if needed.
> 
> That works fine, to be clear, the "version" in the URL, currently "0" should
> only be changed by hand though, right? Or should it increment each time
> there is a new revision of the file?

Yes, currently the URL version is always "0" regardless of the version of the file. Thanks!

Comment 40

[(inactive) Jim Chen [:jchen] [:darchons]]

Hi Brandon, is the cron job running for the override list?

There has been a change to ua-update.json.in a couple weeks ago, https://hg.mozilla.org/mozilla-central/raw-file/tip/b2g/app/ua-update.json.in

But I see the online file hasn't been updated yet, https://dynamicua.cdn.mozilla.net/0/%7B3c2e2abc-06d4-11e1-ac3b-374f68613e61%7D

Comment 41

[Brandon Burton [:solarce]]

(In reply to Jim Chen [:jchen :nchen] from comment #40)
> Hi Brandon, is the cron job running for the override list?
> 
> There has been a change to ua-update.json.in a couple weeks ago,
> https://hg.mozilla.org/mozilla-central/raw-file/tip/b2g/app/ua-update.json.in
> 
> But I see the online file hasn't been updated yet,
> https://dynamicua.cdn.mozilla.net/0/%7B3c2e2abc-06d4-11e1-ac3b-
> 374f68613e61%7D

My apologies, the cron job had a typo in it but the output was getting eaten.

I've fixed the cron job and the latest file has been pushed

Unless you see any issues, we should be good

Comment 42

[Karl Dubost💡 :karlcow]

(In reply to Lawrence Mandel [:lmandel] from comment #21)
> (In reply to Jim Chen [:jchen :nchen] from comment #19)
> > Setting "general.useragent.updates.enabled" to false will disable updating
> > overrides and also disable applying updated overrides (overrides through
> > prefs are independent and are not affected)
> 
> Karl - Can you please include these prefs somewhere in the compat docs.


Done. 
https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Platform/Settings_list$compare?to=486653&from=439411

Plus questions to https://groups.google.com/forum/#!topic/mozilla.engagement.developers/GnrNdxO-zwQ

Comment 43

[(inactive) Jim Chen [:jchen] [:darchons]]

(In reply to Brandon Burton [:solarce] from comment #41)
> (In reply to Jim Chen [:jchen :nchen] from comment #40)
> > Hi Brandon, is the cron job running for the override list?
> > 
> > There has been a change to ua-update.json.in a couple weeks ago,
> > https://hg.mozilla.org/mozilla-central/raw-file/tip/b2g/app/ua-update.json.in
> > 
> > But I see the online file hasn't been updated yet,
> > https://dynamicua.cdn.mozilla.net/0/%7B3c2e2abc-06d4-11e1-ac3b-
> > 374f68613e61%7D
> 
> My apologies, the cron job had a typo in it but the output was getting eaten.
> 
> I've fixed the cron job and the latest file has been pushed
> 
> Unless you see any issues, we should be good

No worries! Looks great now. Thanks!